Papers with full-parameter model
Finding the Dominant Winning Ticket in Pre-Trained Language Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on pre-trained language models show that they can fine-tune parameters but achieve good downstream performance. |
| Approach: | They find that a dominant winning ticket takes up 0.05% of the parameters and is transferable across different tasks. |
| Outcome: | The proposed model can achieve comparable performance with the full-parameter model, the authors show . the dominant winning ticket takes up 0.05% of the parameters, and the model is transferable across tasks, they show - the authors conclude . |
TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing compression approaches remove entire experts, disrupting routing topology and harming performance, or rely on unstructured weight pruning with limited practical efficiency. |
| Approach: | They propose a structured **T**rapezoidal **E**xpert **N**euron **P**running framework that uses a trapezoidal pattern to identify and retain important experts while applying expert neuron pruning (ENP) to less important experts. |
| Outcome: | The proposed framework outperforms the full-parameter model by 10% on code generation tasks under a sparse activation of experts and a 40% routing sparsity. |